Evaluation Metric

Choosing a good evaluation metric for our problem is important. How accurate our estimate is depends on what metric we are using, and depending on the situation, different metrics can be the most suitable.

Mean Absolute Error

If we say our accuracy is 88%, what does it mean to the decision makers? How will they decide how much production to plan for and how much inventory to maintain with that accuracy?

Instead, if we say our forecast error is 100 tonnes, the decision maker will get a concrete number to base his decisions. If he is worried about understocking, for example, he will decide to plan for 100 tonnes more than what the forecast says.

Thus, I decided upon Mean Absolute Error as the most suitable metric for comparing with IBP and to be displayed in the dashboard.

R2 score

Nonetheless, a percentage accuracy as a confidence level can be valuable, as it can help compare our forecast accuracy across different regions and brands/SKUs. Some brands may be very high selling and even if they have a very good accuracy, may have very high errors.

For this reason, we decided to use r2 score to compare forecasts across different brands and regions.